Skip to content

W2-1: sync/index config — ingested sortAt + model3dId, per-image Post fan-out - #326

Merged
JustMaier merged 1 commit into
mainfrom
feat/sync-config-sortat
Jul 17, 2026
Merged

W2-1: sync/index config — ingested sortAt + model3dId, per-image Post fan-out#326
JustMaier merged 1 commit into
mainfrom
feat/sync-config-sortat

Conversation

@JustMaier

Copy link
Copy Markdown
Contributor

Stack

Stacked on #325 (feat/per-image-fanout) — this PR targets feat/per-image-fanout, not main. It depends on #325's fan_out_per_row / shared_ops_fields trigger_gen types and will need to be re-targeted at main (or merged after) once #325 lands. Do not merge before #325.

Implements W2-1 (#8) of docs/design/scheduled-publish-execution-plan.md.

Summary

  • deploy/configs/sync-config-civitai.yaml:
    • Post trigger: type: fan_outfan_out_per_row (rows_from: {table: Image, fk: postId}), retaining track_fields: {publishedAt, availability, postedToId} ([PR-M1] — publishedAt drives both deferred-alive activation stamping and the isPublished shadow; only Phase 2 ever removes availability/postedToId).
    • Image trigger: + sortAtUnix via shared_ops_fields (milliseconds, * 1000, COALESCE(GREATEST(...)) belt — byte-identical to W1-1's SORTATUNIX_EXPR), + model3dId computed_field (INSERT-read Post subselect; not carried on the Post fan-out — model3dId is set at post-create before images exist, so INSERT-read alone suffices; reasoning documented inline in the config).
    • Dump images phase: copy_query gains a type-correct sortAt epoch-seconds column ([PR-B3] — all GREATEST/COALESCE operands are TIMESTAMPs, extract(epoch from ...)::bigint wraps the whole expression, never COALESCE(timestamp, bigint)); posts enrichment gains the model3dId column + field mapping.
  • deploy/configs/civitai-index.yaml:
    • sortAt sort field: removed the computed: {op: greatest, source_fields: [existedAt, publishedAt]} block ([PR-M5] — same-deploy as the emission flip; computed and ingested must never coexist).
    • + model3dId filter field (single_value, per_value_lazy) + data_schema entry.
    • Verified sortAtUnix → sortAt data_schema mapping intact (ms_to_seconds: true, fallback: sortAt).
  • Deleted stale deploy/configs/prod-sync-config-civitai.yaml — behind the canonical sync-config-civitai.yaml since the 2026-07-07 Mode B fix; prod-sync.toml mounts sync-config-civitai.yaml, and grep found no code/operational reference to the stale copy.
  • Regenerated docs/design/trigger-sql-review.sql (test-generated artifact).

[PR-B4] Units verification (must-verify before cutover)

Pinned statement (also written into both config files as comments):

sortAtUnix is emitted in MILLISECONDS; the data_schema sortAtUnix → sortAt mapping (ms_to_seconds: true) divides by 1000, so the sortAt SORT LAYERS store SECONDS. The dump emits a sortAt column already in SECONDS via fallback: sortAt (a fallback source is NOT ms-converted). A query sortAtUnix Gte <seconds> clause resolves ONLY through time_buckets (filter_field: sortAtUnix → sort_field: sortAt): the threshold is unit-normalized by ts_secs_u64 (already seconds, passes through unchanged) and snapped to a bucket built from the sortAt seconds layers. No ingested sortAtUnix filter field exists. Seconds everywhere on the query side; ms only on the wire. The Civitai builder's seconds-Gte (image.service.ts:3940) is CORRECT both before and after this change.

Evidence (file:line):

  • (a) Which field the clause hits: the sort layers via time-bucket, not a filter field — src/executor.rs:668-690 (Gt/Gte bucket-snap gated on field == tb.field_name()), src/cache_worker.rs:575 (bucket membership derived from tb.sort_field_name() = "sortAt").
  • (b) Units: seconds — src/time_buckets.rs:189-196 (cutoff/now arithmetic in seconds), threshold normalized at src/query.rs:316 (ts_secs_u64).
  • (c) Does the period filter work TODAY (sortAtUnix emitted by nothing): YES — buckets are populated from the sortAt sort field (today: engine-computed via GREATEST), the sortAtUnix name is only the clause label matched against tb.field_name(). Whether sortAtUnix is ingested is irrelevant to the query path.
  • (d) After our change (emission ms → mapping ÷1000 → layers seconds): unchanged — sortAt seconds still feed the layers (now ingested instead of computed), seconds-Gte remains correct.
  • Config refs: civitai-index.yaml time_buckets block (filter_field: sortAtUnix, sort_field: sortAt) + data_schema sortAtUnix → sortAt entry.

Caveat (pre-existing, unchanged by this PR): a non-snappable sortAtUnix Gte (duration outside the 10% tolerance of every configured bucket) falls through to range_scan("sortAtUnix"), which returns FieldNotFound since no such filter field is ingested. In practice the Civitai builder only sends period thresholds aligned to the 24h/7d/30d/1y buckets, so this hasn't surfaced — flagging for W3/W4 awareness, not fixing here (out of W2-1 scope).

Tests

src/pg_sync/sync_config.rs, all loading the real deploy configs:

  • test_post_fanout_disjoint_from_sortatunix — Post-fan-out ∩ Image-trigger emitted fields == exactly the intentional Mode-B overlap {publishedAt, availability, postedToId}; sortAtUnix/model3dId strictly disjoint (the actual op_dedup LIFO hazard named in [PR-m2]).
  • test_image_sortatunix_emitted_ms_on_deploy_config — ms units + shared-function call present on the real config.
  • test_index_config_sortat_ingested_and_model3did — no computed: block, mapping semantics, model3dId wiring.

cargo test --lib --features pg-sync (skipping test_min_tracked_value_after_expansion, known deadlock): 1249 passed, 8 ignored, 0 failed.

Notes for reviewers

  • The literal W1-1-review-minor-3 disjointness wording ("Post fan-out track_fields ∩ Image trigger emitted fields = ∅") is unsatisfiable given PR-M1's retained payload — confirmed with the plan coordinator that the implemented invariant (overlap == exactly the Mode-B set; sortAtUnix/model3dId disjoint) is the intended reading.
  • model3dId as a Post column confirmed by the coordinator against prisma/schema.prisma in model-share (model3d-visible-ids-batch.test.ts:6).
  • Does not touch talos-infra (mirroring happens at W4 deploy, per plan).

🤖 Generated with Claude Code

… fan-out

Wires the scheduled-publish sync/index config to the W1-1 (#325)
fan_out_per_row + shared_ops_fields trigger types.

sync-config-civitai.yaml:
- Post trigger: type fan_out -> fan_out_per_row (rows_from Image/postId),
  RETAINING track_fields {publishedAt, availability, postedToId} [PR-M1].
- Image trigger: + sortAtUnix track_field via shared_ops_fields (ms, `* 1000`,
  COALESCE(GREATEST...) belt — byte-identical to W1-1 SORTATUNIX_EXPR);
  + model3dId computed_field (INSERT-read Post subselect, NOT on the Post
  fan-out — set at post-create, INSERT-read suffices; documented inline).
- Dump images phase: copy_query gains a type-correct `sortAt` epoch-SECONDS
  column [PR-B3] (all-timestamp COALESCE/GREATEST, extract epoch ::bigint
  outside); posts enrichment + model3dId column + field mapping.

civitai-index.yaml:
- sortAt sort field: drop `computed:` block ([PR-M5], same-deploy as emission
  flip); + model3dId filter field (single_value, per_value_lazy) + data_schema
  entry; verified sortAtUnix->sortAt mapping (ms_to_seconds + fallback:sortAt).
- [PR-B4] pinned unit statement in both configs.

Tests (src/pg_sync/sync_config.rs, load the REAL deploy configs):
- test_post_fanout_disjoint_from_sortatunix: Post∩Image emitted fields == the
  intentional Mode-B set {publishedAt, availability, postedToId}; sortAtUnix and
  model3dId strictly disjoint (op_dedup safety).
- test_image_sortatunix_emitted_ms_on_deploy_config: ms units + shared fn call.
- test_index_config_sortat_ingested_and_model3did: no computed, mapping + model3dId.

Also: delete stale deploy/configs/prod-sync-config-civitai.yaml (behind the
canonical sync-config-civitai.yaml since the 2026-07-07 Mode B fix; prod-sync.toml
mounts sync-config-civitai.yaml, no code/operational reference to the stale copy).
Regenerated docs/design/trigger-sql-review.sql.

cargo test --lib --features pg-sync: 1249 passed, 8 ignored
(test_min_tracked_value_after_expansion skipped — known deadlock).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@JustMaier
JustMaier force-pushed the feat/sync-config-sortat branch from d567bd9 to 7bff7be Compare July 17, 2026 01:18
@JustMaier
JustMaier changed the base branch from feat/per-image-fanout to main July 17, 2026 01:18
@JustMaier
JustMaier merged commit 4f59d67 into main Jul 17, 2026
5 checks passed
JustMaier added a commit that referenced this pull request Jul 17, 2026
…ates PASS) (#329)

* test(w3): scheduled-publish + ingested-sortAt E2E + full-scale proof harness

Pins the W3 critical cases (scheduled-publish-design §2.D, #325+#326):
- tests/e2e/e2e-scheduled-publish-sortat.mjs — self-contained (no PG) Node E2E
  driving /ops with the exact #326 trigger payloads: ingested sortAtUnix(ms)->sortAt(s),
  scheduled->deferred, the CRITICAL no-op sweep activation at Tf (ingested future sortAt
  preserved through activation), unpublish/republish. 14/14 PASS.
- tests/w3/ — full-scale reproducible harness (prep/dedup/dump-driver/assert, local-PG
  schema + model-share sortAt triggers, minimal poller, lifecycle, latency, re-emitter)
  + README with the gate evidence (all gates PASS: 114.8M dump 23m13s no-regression,
  sortAt invariant 600/600, model3dId 108,567, lifecycle incl. no-op sweep, heal-path,
  latency, PR-B3).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(w3): post-#328 amendments — gate1 valid vs pure-GREATEST, stale-column self-heal, widened [PR-M2]

- README: gate 1 remains valid without re-dump (prep computes pure GREATEST, never reads Image.sortAt
  — identical to #328's amended copy_query). Emptied-originals + stage_dir isolation warnings.
- w3-stale-column.sql: proves the widened BEFORE trigger heals a stale sortAt column — an
  nsfwLevel-only update emits the CORRECT recomputed sortAtUnix (verified: stale 2020 -> correct).
- w3-m2-latency.sql: widened-trigger per-update overhead +47us P50 / +390us P99 (sub-ms).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
JustMaier added a commit that referenced this pull request Jul 17, 2026
…ortAt, model3dId (#332)

* docs: scheduled-publish wrap-up — per-image fan-out, ingested sortAt, model3dId (#13)

Documents the shipped scheduled-publish effort (PRs #325/#326/#328/#329/#330)
across the doc set:

- pg-sync-v2-final.md: Post fan-out is now per-image materialized
  (type: fan_out_per_row) not queryOpSet; shared PG functions
  bitdex_post_fanout_ops / bitdex_image_sortat_ops; sortAt ingested (computed:
  dropped), dump never trusts the column; ModelVersion/Model stay queryOpSet.
- CLAUDE.md: Sync V2 section reflects new fan-out shape + ingested sortAt +
  model3dId.
- scheduled-publish-design.md: §1 history corrected (2024 migration triggers,
  Post-side neutered by name-collision, new_image_sort_at absent from prod
  pg_trigger per zuri's live verification); §2.D marked EXECUTED with the
  no-backfill decision and W1-4 dominant-class (visible-but-misordered) finding.
- execution-plan: W4 step 2 no-backfill correction note.
- FOLLOWUP.md: 7 residuals (over-cap fan-out skip, flaky flush tests, retired
  combinedNsfwLevel, frankenstein docs, MV cascade cost, stage_dir isolation,
  E2E population gap).
- HANDOFF.md: scheduled-publish design chain pointer + W4 rollout state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(reemitter): correct Tf-activation model to sweep-driven shadow flip

Prod evidence (sortat-divergence-specimens-2026-07-16.md PR-m5 + execution
plan corrected [PR-B2]) established that scheduled slots are ALIVE with
isPublished=false (quarantine branch), NOT deferred-map-resident, so Tf
activation is an overdue-sweep shadow flip, not activate_due. The re-emitter is
therefore a co-driver of activation, not a pure safety net. W3 confirmed the
no-op case (sweep alone flipped isPublished + preserved ingested sortAt, 14/14
E2E). Fixes fact #1 + the §7 'does not drive activation' bullet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
JustMaier added a commit that referenced this pull request Jul 17, 2026
… ingested sortAt + model3dId, #328 dump no-backfill, #329 W3 full-scale E2E)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant